Resampling Methods in Software Quality Classification

In the presence of a number of algorithms for classi ̄cation and prediction in software engineering, there is a need to have a systematic way of assessing their performances. The performance assessment is typically done by some form of partitioning or resampling of the original data to alleviate biased estimation. For predictive and classi ̄cation studies in software engineering, there is a lack of a de ̄nitive advice on the most appropriate resampling method to use. This is seen as one of the contributing factors for not being able to draw general conclusions on what modeling technique or set of predictor variables are the most appropriate. Furthermore, the use of a variety of resampling methods make it impossible to perform any formal metaanalysis of the primary study results. Therefore, it is desirable to examine the in°uence of various resampling methods and to quantify possible di®erences.Objective and method: This study empirically compares ̄ve common resampling methods (hold-out validation, repeated random sub-sampling, 10-fold cross-validation, leave-one-out cross-validation and nonparametric bootstrapping) using 8 publicly available data sets with genetic programming (GP) and multiple linear regression (MLR) as software quality classi ̄cation approaches. Location of (PF, PD) pairs in the ROC (receiver operating characteristics) space and area under an ROC curve (AUC) are used as accuracy indicators. Results: The results show that in terms of the location of (PF, PD) pairs in the ROC space, bootstrapping results are in the preferred region for 3 of the 8 data sets for GP and for 4 of the 8 data sets for MLR. Based on the AUC measure, there are no signi ̄cant di®erences between the di®erent resampling methods using GP and MLR. Conclusion: There can be certain data set properties responsible for insigni ̄cant di®erences between the resampling methods based on AUC. These include imbalanced data sets, insigni ̄cant predictor variables and high-dimensional data sets. With the current selection of data sets and classi ̄cation techniques, bootstrapping is a preferred method based on the location of (PF, PD) pair data in the ROC space. Hold-out validation is not a good choice for comparatively smaller data sets, where leave-one-out cross-validation (LOOCV) performs better. For comparatively larger data sets, 10-fold cross-validation performs better than LOOCV.